Skip to content

nvml: Update getDeviceFieldValue to check value of nvmlReturn and use correct valueType - #564

Merged
dbarry9 merged 1 commit into
icl-utk-edu:masterfrom
Treece-Burgess:02-16-2026-nvml-use-correct-value-type
Jun 16, 2026
Merged

nvml: Update getDeviceFieldValue to check value of nvmlReturn and use correct valueType#564
dbarry9 merged 1 commit into
icl-utk-edu:masterfrom
Treece-Burgess:02-16-2026-nvml-use-correct-value-type

Conversation

@Treece-Burgess

@Treece-Burgess Treece-Burgess commented Feb 16, 2026

Copy link
Copy Markdown
Contributor

Pull Request Description

This PR updates the function getDeviceFieldValue to:

  1. Properly check the value of the member variable nvmlReturn as it must be checked before looking at value as value is undefined if nvmlReturn != NVML_SUCCESS.
  2. Account for the different valueType as there are multiple:
NVML_VALUE_TYPE_DOUBLE = 0
NVML_VALUE_TYPE_UNSIGNED_INT = 1
NVML_VALUE_TYPE_UNSIGNED_LONG = 2
NVML_VALUE_TYPE_UNSIGNED_LONG_LONG = 3
NVML_VALUE_TYPE_SIGNED_LONG_LONG = 4
NVML_VALUE_TYPE_SIGNED_INT = 5
NVML_VALUE_TYPE_UNSIGNED_SHORT = 6

Due to not checking the valueType there were cases when extremely large numbers would output for nvml:::NVIDIA_A100-PCIE-40GB:device_0:gpu_inst_power:

[ICL:methane bin]$ ./papi_command_line nvml:::NVIDIA_A100-PCIE-40GB:device_0:gpu_inst_power

This utility lets you add events from the command line interface to see if they work.

Successfully added: nvml:::NVIDIA_A100-PCIE-40GB:device_0:gpu_inst_power

nvml:::NVIDIA_A100-PCIE-40GB:device_0:gpu_inst_power : 	4991201436214789335 mW

Testing

Setup

Testing was done on Methane at ICL with:

  • OS: RHEL 9.6
  • CPU: Intel Xeon Gold 6140
  • GPU: 1 * A100
  • Cuda Toolkit: 12.9

Results

  • PAPI build: ✅
  • PAPI utilities*: ✅
# Sanity checking nvml:::NVIDIA_A100-PCIE-40GB:device_0:gpu_inst_power via 
# papi_command_line did not show an extremely large value after 10 runs:

[ICL:methane bin]$ ./papi_command_line nvml:::NVIDIA_A100-PCIE-40GB:device_0:gpu_inst_power

This utility lets you add events from the command line interface to see if they work.

Successfully added: nvml:::NVIDIA_A100-PCIE-40GB:device_0:gpu_inst_power

nvml:::NVIDIA_A100-PCIE-40GB:device_0:gpu_inst_power : 	33925 mW

  • nvml HelloWorld.cu test: ✅

* - papi_component_avail, papi_native_avail, and papi_command_line

Author Checklist

  • Description
    Why this PR exists. Reference all relevant information, including background, issues, test failures, etc
  • Commits
    Commits are self contained and only do one thing
    Commits have a header of the form: module: short description
    Commits have a body (whenever relevant) containing a detailed description of the addressed problem and its solution
  • Tests
    The PR needs to pass all the tests

@Treece-Burgess Treece-Burgess added type-bug Issues discussing bugs or PRs fixing bugs component-nvml PRs and Issues related to the nvml component status-ready-for-review PR is ready to be reviewed labels Feb 16, 2026
@dbarry9

dbarry9 commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

I'm reviewing this PR.

@dbarry9
dbarry9 force-pushed the 02-16-2026-nvml-use-correct-value-type branch from 6cd066a to 9d48fd3 Compare June 15, 2026 18:08
@dbarry9

dbarry9 commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

I tested these changes on the NVIDIA GH200 480GB device (Grace-Hopper architecture).

The utility papi_command_line indeed reports reasonable event counts:

~/papi/src$ papi_command_line nvml:::NVIDIA_GH200_480GB:device_0:gpu_inst_power

This utility lets you add events from the command line interface to see if they work.

Successfully added: nvml:::NVIDIA_GH200_480GB:device_0:gpu_inst_power

nvml:::NVIDIA_GH200_480GB:device_0:gpu_inst_power : 	112654 mW

----------------------------------

However, the nvml component test HelloWorld returns the following error:

~/papi/src$ ./components/nvml/tests/HelloWorld
Call to cudaGetDeviceCount(&num_devices) on line 148 failed with error code 805.

@Treece-Burgess Does this error occur on your end?

@Treece-Burgess

Copy link
Copy Markdown
Contributor Author

I tested these changes on the NVIDIA GH200 480GB device (Grace-Hopper architecture).

The utility papi_command_line indeed reports reasonable event counts:

~/papi/src$ papi_command_line nvml:::NVIDIA_GH200_480GB:device_0:gpu_inst_power

This utility lets you add events from the command line interface to see if they work.

Successfully added: nvml:::NVIDIA_GH200_480GB:device_0:gpu_inst_power

nvml:::NVIDIA_GH200_480GB:device_0:gpu_inst_power : 	112654 mW

----------------------------------

However, the nvml component test HelloWorld returns the following error:

~/papi/src$ ./components/nvml/tests/HelloWorld Call to cudaGetDeviceCount(&num_devices) on line 148 failed with error code 805.

@Treece-Burgess Does this error occur on your end?

This error occurs even in the sample code /packages/cuda/13.0.0/extras/CUPTI/samples/concurrent_profiling/concurrent_profiling.cu provided by NVIDIA on Hopper1 at Oregon (1 * GH200):

[tburgess@hopper1 concurrent_profiling]$ ./concurrent_profiling 


Error: concurrent_profiling.cu:218: Function cudaGetDeviceCount(&numDevices) failed with error(805): MPS client failed to connect to the MPS control daemon or the MPS server.

[tburgess@hopper1 concurrent_profiling]$ 

If I go to Illyad on Oregon (1 * H100) with Cuda Toolkit 13.0.0, this error does not occur:

[tburgess@illyad tests]$ ./HelloWorld 
Running the nvml component test HelloWorld.cu
PAPI version being used for this test: 7.3.0
The nvml component is assigned to component index: 2

Starting string:
Hello World!

Proceeding to mangle the starting string.
The mangled string is: HdjikhjcZ

Proceeding to unmangle the mangled string.
The unmangled string is: Hello World!

After PAPI_stop, the event nvml:::NVIDIA_H100_PCIe:device_0:graphics_clock produced the value 		1755

This appears to be a system issue rather than a PAPI issue.

@dbarry9
dbarry9 force-pushed the 02-16-2026-nvml-use-correct-value-type branch from 9d48fd3 to 878ea4f Compare June 16, 2026 14:42
@dbarry9
dbarry9 merged commit 9bed387 into icl-utk-edu:master Jun 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component-nvml PRs and Issues related to the nvml component status-ready-for-review PR is ready to be reviewed type-bug Issues discussing bugs or PRs fixing bugs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants